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PREFACE 


This  report  was  created  for  the  F— 1 6  Aircrew  Training 
Development  Project  contract  no.  F02604-79-C8875  for  the  Tactical 
Air  Command  to  comply  with  the  requirements  of  CDRL  no.  B007. 

The  project  entailed  the  design  and  development  of  an  instruc¬ 
tional  system  for  the  F16  RTU  and  instructor  pilots.  During  the 
course  of  the  project,  a  series  of  development  reports  was  issued 
describing  processes  and  products.  A  list  of  those  reports 
follows  this  page.  The  user  is  referred  to  Report  No.  34,  A 
Users  Guide  to  the  F-16  Training  Development  Reports,  for  an 
overview  and  explanation  of  the  series,  and  Report  No.  35,  F-16 
Final  Report,  for  an  overview  of  the  Instructional  System 
Development  Project. 
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F-16  AIRCREW  TRAINING 
DEVELOPMENT  PROJECT  REPORTS 


Copies  of  these  reports  may  be  obtained  by  writing  the  Defense 

Technical  Information  Center,  Cameron  Station,  Alexandria, 

Virginia  22314.  All  reports  were  reviewed  and  updated  in  March 
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EXECUTIVE  SUMMARY 


This  report  details  the  forms  developed  for  organizing  and 
storing  the  voluminous  ISD  data/information  generated  by  the  F-16 
training  project.  The  forms  are  designed  to  accommodate  continu¬ 
ally  changing  data  relative  to  hundreds  of  tasks,  instructional 
objectives,  and  production  elements.)  Although  the  forms  are  the 
product  of  experience  in  manually  operated  systems,  they  are 
compatible  with  computer  based  d-aTfa  systems.  The  formats  used 
have  been  changed  and  modified  as  necessary  as  F-16  ISD  pro¬ 
cedures  have  changed. 

^  This  report  presents  a  sample  and  description  of  the  forms 
used  in  the  analysis  and  design  phases  of  the  program,  that  is 
XT)  tasks  listing  and  objectives  hierarchy  analysis  (task 
T.i sting/objectives  hierarchy  worksheet  and  task  specification 
worksheet),'  (gl  media  selection  (hands-on  media  selection  forms, 
media  by  capability  matrices,  media  priority  by  segment  sheet, 
and  media  selection  tally  sheet)  ,'  £3")  personnel  management 
(weekly  time  summary  sheet)  ,’  (4f)  production  management  (pro¬ 
duction  management  forms,  weekly  progress  report,  and  production 
tracking  chart)  ;  and  ^fr)  pre- implementation  evaluation  (student 
attitude  questionnaire)  . 

The  fact  that  the  forms  illustrated  and  described  were  de¬ 
signed  and  redesigned  during  the  F-16  project  is  not  of  concern, 
since  forms  are  management  tools  to  serve  the  thought  processes, 
and  not  vice  versa.)  Tools  in  any  industry  are  designed/re¬ 
designed  to  meet^tbe  changing  requirements  of  that  industry. 

^Despite  their  susceptibility  to  change,  it  is  recommended 
that  great  care  be  taken  in  providing  the  data  called  for  by  the 
forms  because  they  are  the  product  of  experience  gained  from 
other  ISD  projects.  Also,  by  early  development  of  these  forms, 
data  collection  was  timely  and  systematic,  resulting  in  the 
savings  of  time  and  money. 

Data  bases  required  for  an  automated  ISD  management  system 
have  all  been  defined,  should  automation  be  implemented  after  the 
contractor  has  left  the  F-16  training  program. 
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DATA  COLLECTION  AND  MANAGEMENT 
FORMS  REPORT 


1.0  IHTRODUCTION 


This  report  is  in  response  to  a  contract  requirement  to 
specify  data  collection  and  data  management  forms  to  be  used  to 
support  development  of  F-16  pilot  training.  The  mountains  of 
data  generated  during  a  project  of  the  F-16's  scope  require 
extreme  care  in  collection  and  handling.  Scores  of  continually 
changing  data  items  relative  to  hundreds  of  tasks,  instructional 
objectives,  and  production  elements,  must  be  recorded,  accessed, 
updated,  and  summarized.  The  forms  contained  in  this  report  are 
designed  for  maximum  usefulness  in  support  of  these  data  manage¬ 
ment  requirements.  The  forms  are  the  product  of  experience  in 
manually  operated  data  systems. 

Many  different  data  forms  are  employed  during  the  life  of  an 
ISD  project.  The  need  for  them  arises  directly  out  of  the  proc¬ 
esses  they  support.  Because  of  this,  the  structure  of  this 
report  roughly  conforms  to  the  ISD  processes.  It  is  divided  into 
the  following  major  parts: 


Task  Listing/Objectives  Hierarchy  Analysis  Forms 

Media  Selection  Forms 

Personnel  Management  Forms 

Production  Management  Forms 

Evaluation  Before  Implementation  Forms 

The  forms  developed  and  used  by  the  project  are  provided  and 
discussed  in  two  development  reports:  Data  collection  and 
management  forms  report  (# 3)  and  F-16  implementation  and 
management  plan  report  (#18) .  This  report  includes  the  forms 
used  during  the  analysis,  design  and  development  phases  of  the 
program,  prior  to  course  implementation,  while  report  #18 
includes  those  forms  used  for  course  implementation,  evaluation 
and  revision. 

It  is  should  be  pointed  out  that  no  matter  how  well  designed 
a  set  of  forms  and  management  aids  may  be,  they  still  will  need 
to  be  used  with  care  and  thoughtfulness.  The  clearest  data 
collection  form  cannot  collect  its  own  data.  Nor  can  the  most 
efficient  management  scheme,  on  paper  or  on  computer,  drive 
itself.  The  data  forms  described  herein  will  be  used  as  a  tool 
by  the  developers  for  capturing  and  systematizing  the  decisions, 
information,  and  creative  ideas  which  evolve  during  the 
development  of  the  F-16  training  system. 
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2.0  TASK  LISTING  AND  OBJECTIVES  HIERARCHY  ANALYSIS 


During  task  listing,  an  inventory  of  the  tasks  to  be  trained 
was  developed,  and  conditions,  standards  and  appropriate  refer¬ 
ence  materials  specified.  Once  the  tasks  to  be  learned  were 
identified  and  specifications  developed,  each  task  was  subjected 
to  objectives  hierarchy  analysis  to  yield  a  set  of  instructional 
objectives  that  build  up  student  behavior  in  a  progression  of 
steps  required  to  accomplish  performance  objectives.  Through 
this  process,  the  required  skills  and  knowledges  were  identified 
which  were  necessary  prerequisites  to  mastery  of  the  performance 
objectives.  The  end  result  of  the  process  was  a  series  of  maps 

illustrating  the  order  in  which  objectives  build  on  one  another 
(i.e.,  their  hierarchical  relation). 

The  forms  developed  were  a  Task  Listing/Objectives  Hierarchy 
Worksheet  and  a  Task  Specification  Worksheet. 
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2.1  TASK  LISTING/OBJECTIVES  HIERARCHY  WORKSHEET 


The  Task  Listing/Objectives  Hierarchy  Worksheets  were  con¬ 
structed  by  SME/Writer  teams.  The  process  started  with  a  higher 
order  task,  i.e.,  perform  a  MIL/MAX  AB  Power  Departure,  followed 
by  the  necessary  sub-tasks,  i.e.,  perform  Level-off.  The  end 
result  was  a  combination  task  listing  and  objectives  hierarchy 
portrayed  as  a  series  of  connected  boxes  which  demonstrated  the 
behaviors  necessary  to  accomplish  the  task. 

The  behavior  statement  goes  in  the  top  part  of  each  box  and 
a  reference  number  goes  in  the  bottom  part.  The  representation 
system  for  the  objectives  hierarchies  is  described  below. 

This  symbol  is  used  to  indicate 
the  objective  that  is  immediately 
superordinate  to  the  objectives 
on  a  given  hierarchy  page.  This 
is  done  to  orient  the  reader  to 
the  location  of  the  superordinate 
objectives  in  the  total 
objectives  hierarchy. 


2.  This  symbol  is  used  to  indicate 
an  objective  analyzed  on  a  hier¬ 
archy  page. 


I - 1  1. 

I  I 

I  I 

I - 1 

I _ I 


3.  The  hex  box  is  used  to  indicate 
an  objective  which  is  further 
analyzed  on  a  subsequent  hier¬ 
archy  diagram  page.  An  objective 
so  denoted  will  be  the  first 
solid  line  rectangular  box  on  the 
subsequent  hierarchy  diagram 
page. 


2.2  TASK  SPECIFICATION  WORKSHEET 

The  Task  Specification  Worksheet  is  headed  with  the  task 
number  and  behavior,  i  .e.  ,  task  number  1.1.5,  Perform  Mission 
Briefing  (flight  lead).  The  information  to  be  filled  in  (by 
SMEs)  included  conditions,  standards,  a  description  of  required 
steps  and  reference  materials.  The  information  contained  was 
used  to  generate  computer  printouts  of  each  task  and  its  specifi¬ 
cations  . 
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FIGURE  2.1  TASK  LISTING/OB JE<~  1 1 
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FIGURE  2.2  TASK  SPECIFICATION  WORKSHEET 
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3.0  MEDIA  SELECTION 

Once  all  the  learning  objectives  have  been  defined,  the  most 
appropriate  media  for  teaching  each  objective  must  be  determined, 
as  well  as  acceptable  alternate  media.  This  information  is  used, 
along  with  scheduling  and  resources  information,  to  determine  the 
final  media  mix  for  the  syllabus.  The  results  of  this  study  can 
also  have  a  minor  influence  to  the  sequencing  of  learning  objec¬ 
tives  in  the  syllabus.  The  media  selection  process  was  carried 
out  for  both  hands-on  objectives  and  academic  objectives.  The 
forms  used  consisted  of  media  matrices,  an  Academic  Media  Selec¬ 
tion  Form,  a  Media  Priority  by  Segment  Sheet,  and  a  Media  Selec¬ 
tion  Tally  Sheet.  For  the  hands-on  objectives  two  matrices  were 
utilized:  A  Device  by  Capability  Matrix  and  a  Hands-on  Objective 
by  Media  Characteristics  Matrix.  The  academic  matrix  used  was  a 
Media  by  Capability  Matrix. 
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3.1  HANDS-ON  MEDIA  SELECTION  FORMS 


» 

_k. 


The  Device  by  Capability  Matrix  was  designed  with  the  devices, 
i.e.,  panel  mockup  1:1,  listed  along  the  top  row  and  the 
capabilities,  i.e.,  physical  layout  characteristics,  conformity 
with  shape,  listed  down  the  left  side.  An  "x"  was  used  to 
indicate  that  a  device  did  possess  a  certain  capability. 

The  Hands-on  Media  Selection  Worksheet  was  designed  with  blank 
columns  along  the  top  (to  be  filled  in  with  the  objective 
numbers)  and  media  characteristics,  i.e.,  physical  layout  charac¬ 
teristics,  conformity  with  shape,  listed  down  the  left  side.  A 
check  matrix  was  used  to  indicate  that  an  objective  did  require  a 
certain  media  characteristic. 

For  a  detailed  discussion  of  the  procedures  used  in  academic 
and  hands-on  media  selection,  refer  to  Report  No.  31*  "F-16 
Training  Media  Mix”. 
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FIGURE  3.1.1  INITIAL  HANDS-ON  MEDIA  SELECTION  MATRIX 
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Between-PaneT'  Spatial  Relations 


Within-Panel  Spatial  Relations 


SlinULUS  fKUl'tKLUib  -  1 

Fidelity  of  Visual  Stimulu 


Fidelity  of  Aud  (non  voice)  Stim 


Fidelity  of  Kinesthetic  Stimulus 


Voice  Stimulus  Capability 


Field  of  View 


Proportion  of  Distant  Obj 


Coloration  and  Shading  -  Obj 


Visual  Sensation  -  Motion 


Shape  of  Objects 


Resolution  -  Detail  Given  Distance 


Fidel?tyI5VLfflnipu?a9?§81,ES _ 


Timely  Manip-Resp  Connection  _ 


RESPONSE  PROPERTIES 
fidelity  of  Interior  Response _ 


Fidelity  of  Exterior  Response 


Stop  Action  Capability 


Canned  Demo  Capability 


Live  Demo  Capability 


Conditions  Reset  Capability 


,  EVALUATION  FEATURES 

R/W  Feedback 


Out  of  Tolerance  Feedback 


Instructor  Feedback 


of  Student  Performanc 


M  _  OPPONENT 

Maneuver  Opponent 


FIGURE  3.1.2  HANDS-ON  MEDIA  SELECTION  WORKSHEET 


3.2  MEDIA  BY  CAPABILITY  MATRICES 


The  Academic  Media  by  Capability  Matrix  was  designed  with 
the  academic  media,  i.e.,  CAI  plus  lesson  guide,  listed  in  the 
top  columns  and  the  academic  media  capabilities,  i.e.,  audio¬ 
voice,  listed  down  the  right  side.  A  rating  scale  of  1  to  5  was 
used.  An  explanation  of  the  scale  follows. 

1.  POOR  -  This  is  the  medium  of  last  resort.  Effectiveness, 
efficency,  scope  and  the  flexibility  are  poor.  There  may  be 
almost  complete  reliance  on  secondary  media  and  there  is  a 
potential  for  negative  affect. 

2.  ADEQUATE  -  This  medium  is  acceptable,  but  weak.  The  scope 
and  flexibility  are  limited  and  both  effectiveness  and 
efficiency  are  poor.  There  may  be  a  strong  dependence  on 
secondary  media. 

3.  GOOD  -  This  medium  will  teach  the  objective  adequately.  It 
is  reasonably  effective,  efficient,  and  flexible.  It  will 
provide  the  most  commonly  required  instructional  instances. 

4.  DESIRABLE  -  This  is  a  good  medium  for  the  objective.  It  is 
efficient,  effective,  and  flexible.  A  broad  range  of 
possible  instructional  instances  are  available,  and  there  is 
potential  for  positive  affect. 

5.  IDEAL  -  This  is  the  best  medium  for  the  objective.  Its 
efficiency,  effectiveness,  and  flexibility  are  outstanding. 
The  scope  is  adequate  for  the  full  range  of  possible  in¬ 
structional  instances  and  there  is  a  likelihood  of  positive 
affect . 

The  hands-on  media  by  capability  matrix  was  designed  in  like 
manner,  but  with  hand-on  media  listed  across  the  top  instead  of 
academic  media.  The  rating  scale  and  method  of  utilization  were 
identical.  This  matrix  updated  and  replaced  the  initial  hands-on 
media  selection  matrix  shown  on  page  number  9. 
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FIGURE  3.2.1  ACADEMIC  MEDIA  BY  CAPABILITY  MATRIX 
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3.3  ACADEMIC  MEDIA  SELECTION  WORKSHEET 


The  Academic  Media  Selection  Worksheet  was  designed  such 
that  the  objective  reference  numbers  were  listed  down  the  left 
side  and  the  instructional  media  requirements,  i.e.,  key  scored 
evaluation,  were  listed  along  the  top.  The  appropriate  boxes 
were  checked  giving  a  profile  of  the  required  instructional  media 
requirements  for  each  objective.  Note  the  parallel  between  this 
form  and  the  hands-on  media  selection  worksheet  shown  on  page 
number  10. 


FIGURE  3.3  ACADEMIC  MEDIA  SELECTION  WORKSHEET 


3.4  MEDIA  PRIORITY  BY  SEGMENT  SHEET 

The  Media  Priority  by  Segment  Sheet  contained  the  segment 
number  (a  segment  consisted  of  the  academic  objectives  necessary 
to  accomplish  a  higher  order  task,  i.e.,  prepare  an  enroute  map); 
the  segment  name,  i.e.,  enroute  mission  planning;  the  near  term 
media  selected  and  its  priority  (indicated  by  a  1  -  5  rating 
scale,  1  indicating  that  the  near  term  media  selected  is  of  the 
highest  priority  and  much  consideration  should  be  given  before 
changing  to  an  alternate  media,  a  5  indicating  a  low  priority  for 
the  near  term  media  and  signifiying  the  near  term  alternate  media 
selected  is  equally  acceptable);  the  near  term  alternate  media 
and  the  ultimate  media. 
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Segment 

Number 

Segment  Name 

MEDIA  ' 

_ i 

Near  Term 

Prior¬ 

ity 

Near  term 
Alternate 

U1  timate 

Comments 

152 

Navigation  Using  Ground 

Map  Radar 

Seminar 

2 

Lecture  - 
R.S.  & 

L.G. 

Tutorial 

W/L.G. 

153 

Radar  Jamming 

Semi nar 

2 

Lecture  - 

Response 

System 

L.G. 

Tutorial 

W/L.G. 

i 

l 

154 

Radar  &  Navigation 

Seminar 

2 

Lecture  - 
R.  Sys.  & 
L.G. 

Tutorial 

W/L.G. 

155 

Basic  G.M.P.  Radar 

Seminar 

2 

Lecture  - 
R.S.  &  L. ( 

Tutorial 
.  W/L.G. 

156 

I.N.S.  Updates 

Seminar 

w/equip 

2 

Lecture  - 
R.S.  & 

L.G. 

CAI 

157 

Radar  Altitude  Calculation 

Semi nar 
w/equip 

2 

Lecture  - 
R.S.  & 

1  G. 

CAI 

1 

! 

i 

158 

A.R.A.  Procedure 

Seminar 

w/L.G. 

2 

Lecture  - 
w/R.S. 

L.G. 

Tutorial 

w/L.G. 

159 

Air  to  Ground  Fence  Check 

Seminar 

w/L.G. 

2 

Lecture 
R.S.  & 

L.G. 

CAI 

160 

Procedure  for  making  T.O.T. 

Seminar 

w/L.G. 

2 

Lecture 
R.S.  & 

L.G. 

CAI 

FIGURE  3.4  MEDIA  PRIORITY  BY  SEGMENT  SHEET 
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3.5  MEDIA  SELECTION  TALLY  SHEET 


The  media  selected  were  tallied  according  to  near  term,  near 
term  alternate  and  ultimate  selections  and  by  phase,  i.e.. 
Conversion,  NAV,  Intercept,  etc.  For  example,  the  Tally  Sheet 
for  BFM  phase,  near  term  media,  might  have  indicated  4  workbooks 
plus  lesson  guide,  3  tapeslides  plus  lesson  guide,  and  15 
seminars  with  visual  motion  and  lesson  guides.  Each  Tally  Sheet 
consisted  of  the  appropriate  heading,  i.e.,  Near  Term,  BFM  and  a 
listing  of  the  32  possible  academic  media. 


Defined  Media  -  Academic 

1  CAI  -  Lesson  guide 

2  CAI  -  Videotape  -  Lesson  guide 

3  CAI  -  Videotape  -  Part  task  trainer  -  Lesson  guide 

4  Interactive  part  task  trainer  -  Lesson  guide 

5  Random  access  slide  -  Lesson  guide 

6  Motion  picture  -  Lesson  guide 

7  Videotape  -  Lesson  guide 

8  Videodisc  -  Lesson  guide 

9  Tape  slide  -  Lesson  guide 

10  Beseler  Cue-See  -  Lesson  guide 

11  Workbook  -  Lesson  guide 

12  Color  workbook  -  Lesson  guide 

13  Workbook  -  Slides  -  Lesson  guide 

14  Workbook  -  Audio  -  Lesson  guide 

15  Programmed  text  -  Lesson  guide 

16  Training  manuals  -  Lesson  guide 

17  Model/actual  equipment  -  Lesson  guide 

18  Cockpit  Familiarization  Trainer  -  Lesson  guide 

19  Cockpit  Familiarization  Trainer  -  Tape  slide  -  Lesson  guide 

20  Lecture  -  Lesson  guide 

21  Lecture  -  Audio  -  Lesson  guide 

22  Lecture  -  Visual  motion  -  Lesson  guide 

23  Lecture  -  Model/actual  equipment  -  Lesson  guide 

24  Tutorial  -  Lesson  guide 

25  Tutorial  -  Audio  -  Lesson  guide 

26  Tutorial  -  Visual  motion  -  Lesson  guide 

27  Tutorial  -  Model/actual  equipment  -  Lesson  guide 

28  Seminar  -  Lesson  guide 

29  Seminar  -  Audio  -  Lesson  guide 

30  Seminar  -  Visual  motion  -  Lesson  guide 

31  Seminar  -  Model/actual  equipment  -  Lesson  guide 

32  Lecture  -  Student  response  system  -  Lesson  guide 


Defined  Media  -  Hands-on 

1  Panel  mockup  1:1 

2  Cockpit  mockup  1:1 

3  Stick  and  throttle  trainer  (automated) 

4  SMS  trainer  (automated) 

5  Avionics  display  (automated) 

6  Active  2-dimensional  isolated  system  trainer 

7  Radar  Warning  Receiver  Trainer  (RWRT) 

8  Interactive  Cockpit  Procedures  Trainer  (ICPT) 

9  Cockpit  Familiarization  Trainer  (CFT) 

10  Egress  Procedures  Trainer  (EPT) 

11  DSS 

12  ASPT 

13  SAAC 

14  Operational  Flight  Trainer  (OFT) 

15  Operational  Flight  Trainer  -  Night  Visual  System  (OFT-NVS) 

16  Operational  Flight  Trainer  -  Digital  Land  Mass  Radar  System  (OFT-DLMRS) 

17  Operational  Flight  Trainer  -  Electronic  Warfare  (OFT-EW) 

18  Weapons  System  Trainer  (WST) 

19  F-16A  Aircraft 

20  F-16B  Aircraft 

21  3D  model  or  actual  equipment 

FIGURE  3.5  ACADEMIC  MEDIA  SELECTION  TALLY  SHEET 
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4.0  PERSONNEL  MANAGEMENT 

A  large-scale  ISD  project  is  in  part  an  evolutionary 
process.  Occurrences  and  analyses  early  in  the  project  may 
determine  later  events.  Personnel  output  must  be  monitored 
during  the  project  to  determine  whether  the  project  is  on 
schedule  and  whether  initial  time  estimates  used  to  calculate 
staffing  levels  were  accurate.  The  data  collected  can  be  used  to 
update  production  scheduling  during  the  project  and  to  allow 
changes  in  scheduling  to  minimize  the  chance  of  bottleneck  later. 
The  data  may  also  be  used  to  reassign  personnel  or  project  per¬ 
sonnel  staffing  level  as  needed. 
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4.1  WEEKLY  TIME  SUMMARY  SHEET 


The  Weekly  Time  Summary  Sheet  indicated  the  number  of  hours 
spent  daily  by  each  instructional  writer  on  individual  segments 
and  the  total  hours  spent  on  all  segments.  The  days  of  the  week 
were  listed  down  the  left  side  with  a  total  at  the  bottom.  These 
were  four  columns  across  the  top  for  four  separate  seg¬ 
ments  (lessons)  and  the  hours  spent  on  each  segment.  As  an 
example,  on  Monday,  2  hours  were  spent  on  WB110,  1  hour  on  AS108 
and  3  hours  on  WB121.  The  total  column  for  the  week  might  in¬ 
dicate  14  hours  were  spent  working  on  WB110.  There  was  also  a 
space  for  total  hours  that  indicated  the  total  hours  spent 
working  on  all  segments. 
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FIGURE  4.1  WEEKLY  TIME  SUMMARY  SHEET 


5.0  PRODUCTION  MANAGEMENT 


During  the  ISD  project,  many  segments  of  instruction  were  in 
different  stages  of  production  at  the  same  time.  It  was  crucial 
that  the  progress  of  each  segment  be  carefully  monitored  through 
the  production  cycle.  An  effective  method  to  accomplish  this  was 
provided  by  the  production  management  forms  described  in  the 
following  section. 


5.1  PRODUCTION  MANAGEMENT  FORMS 

Production  Management  Forms  were  constructed  to  document  and 
facilitate  the  progress  of  workbooks,  tape  (audio)  slides,  work¬ 
book  slides  and  seminars.  The  main  stages  of  progress  documented 
were  the  writer/SME  stages  (outline,  rough  draft,  etc.),  graphic 
stages  (art,  construction  of  dummy  books,  etc.)  and  word  process¬ 
ing  stages.  The  workbook  forms  were  also  used  to  record  the 
progress  of  seminars. 
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TAPE  SLIDE 


Date 

In 


Date 

Out 


Initials 


IW/SME  Produce  Outline 
I  I  Quality  Control  Outline 

Q  Writer  Produces  Draft  Script  and  Visual  Specs 
O  Word  Process  Draft  Script 
0  SME  Reviews  Draft  Script  &  Visual  Specs 
0  Writer  Revises  Script  and  Visual  Specs 
1  I  Quality  Control  Script  and  Visual  Specs 

0  Edit  Script 
0  Produce  Sketches 
0  Narrate  Scratch  Tape 
I  I  IW/SME  Review 

1  I SSME  Review 

ART 

Q  Sketches  sent  to  Hughes  Aircraft 
I  IWriter/SME  Review 

PHOTO 

0SME  Develops  Shooting  Schedule 

0  Graphics  Coordinator  schedules  Photographer  (enter  available  dates/time) 
0  SME  Schedules  aircraft/equipment  (enter  shooting  date/time) 

0  SME  and  Photographer  Take  Pictures 
0  Photos  Delivered 
I  I SME/IW/GC  Review  .  „otos 

f  1  SSME  Pre  Trial  Review 

0  Ed  Spec  Pre  Trial  Review 
0  Conduct  Tryout 
0  Writer  Revisions  Specification 
0  Revise  Script 
0  Revise  Graphics 
0  Re-shoot  Photos 
0  Script  sent  to  Narrator 
I  )Writer/SME  Review 

0  Assemble  Master 

0  Edit  Master _ 

C  I USAF  Acceptance _ 


FIGURE  5.1  PRODUCTION  MANAGEMENT  FORMS  FOR  THE  SLIDE  PROGRAMS 


t  JL 


WORKBOOK 


Date 

In 

Date 

Out 

Initials 

□  Writer/SME  produce  outline 

1  1  Quality  Control  Outline 

0  Writer/SME  produce  draft  (w/visual  specs) 

0  Word  Process  Draft 

0  SME  reviews  draft 

0  Writer  revises  draft 

1  1  Quality  Control  Draft 

1  1  SSME  Review 

0  WP  Semi  Final  Draft 

I  1  Writer/SME  produce  dummv  book 

0  Edit  Semi  Final  Draft 

0  Produce  Graphics-thumbnail 

1  1  Writer/SME  Review 

0  Produce  Final  Graphics 

1  1  Writer/SME  Review  of  Final  Graphics 

1  1  SSME  /  Ed  Spec  Review 

0  Produce  Tryout  Copy  (Graphics) 

0  Conduct  Tryout 

0  Produce  Revision  Specs 

l~)  Revise  Text 

0  Word  Process  Revisions 

0  Revise  Graphics 

i  1  SSME  Final  Review  of  Revisions 

0  Produce  Final  Master 

0  Edit  Master 

f  1  USAF  Acceptance 

FIGURE  5.1.2  PRODUCTION  MANAGEMENT  FORM  FOR  WORKBOOK  PROGRAMS 
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Date 

Date 

In 

Out 

Initials 


□  IW/SME  Produce  Outline 


Quality  Control  Outline 


□  Writer  Produces  Draft  Text  w/Visual  Specs 


0  Word  Process  Draft  Text 


0  SME  Reviews  Draft 


0  Writer  Revises  Draft 


I  Quality  Control  Draft 


SSME  Review 


0  WP  Semi  Final  Draft 


0  Writer/SME  Assembles  dummy  book 


0  Edit  Semi  Final  Draft 


0  Produce  Sketches 


IW/SME  Review 


0  Sketches  Sent  To  Hughes  Aircraft 


Writer/SME  Review 


0  SME  Develops  Shooting  Schedule 


!~~1  Graphics  Coordinator  Schedules  Photographer  (enter  avail,  dates/time) 


n  SME  schedules  Aircraft/Equipment  (enter  shooting  date/time) 


0  SME  and  Photographer  Take  Pictures 


0  Photos  Delivered 


SME/IW/GC  Review  Photos 


SSME/Ed  Spec  Pre  Trial  Review 


0  Conduct  Tryout 


0  Produce  Revision  Specs 


0  Revise  Text 


0  Revise  Graphics 


0  Reshoot  Photos 


0  Assemble  Master 


0  Edit  Master 


USAF  Acceptance 


FIGURE  5.1.3  PRODUCTION  MANAGMENT  FORM  FOR  WORKBOOK/SLIDE 

PROGRAMS 
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5.2  WEEKLY  PROGRESS  REPORT 

The  Weekly  progress  Report  was  designed  to  monitor  the 
weekly  progress  of  individual  lessons,  i  ,e . ,  WB201.  It  was 
divided  into  pnases  (Intercept,  8FM,  etc.)  and  each  lesson  in 
that  phase  was  listed.  The  writer/SME  team  and  the  expected 
class  data  for  each  lesson  was  listed  and  the  current  stage  of 
production  (as  of  each  Friday)  was  indicated.  The  stage  of 
production  was  listed  as  a  step  number  and  a  description  of  that 
step.  The  production  steps  were  the  same  as  those  listed  on  the 
Production  Management  Forms,  i  .e . ,  word  process  draft,  step 
number  4. 
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BFM  PHASE 

_ _  March  18,  1980 

Program  Number/Title  Writer/SME  Stage  of  Program  Class  date 


WB  301  Air-To-Air 

gun  attack  § 
tracking  pro. 

Clark/Hopler 

9  WP  semi-final 
draft 

4/30 

WB  302  Computed  gun 
attack  modes: 

LCOS  5  SS. 

Clark/Roth 

8  SSME  Review 

4/30 

WB  303  Pro.  for 

Air-to-Air  weapons 
system/fence  check. 

Wild/Roth 

9  WP  semi-final 
draft 

5/9 

WB  304  F-16  energy  man. 

§  HUD  energy  mana. 
symbology. 

Foster/Stuart 

9  WP  semi-final 
draft 

5/12 

SM  301  Basic  fighter 
Maneuvers; 

Pursuit 

Guthrie/Kimmel 

9  WP  semi-final 
draft 

5/7 

l 

\ 

i 

SM  302  Basic  fighter 
Maneuvers : 

Principals  § 

Applica. 

Guthrie/Cary 

1 

17  Produce  Tryout  1 
Copy  | 

1 

5/13 

SM  303  Principals 

5  techniques  for 
gun  attack. 

Clark/Hopler 

t 

9  WP  semi-final 
draft 

5/1 

SM  304  Appli.  of  offen 
sive  BFM 

Wild/Clark 

18  Conduct  Tryout 

5/14 

SM  305  Zone  defense  § 

consid.  for  offen 
sive  §  counter¬ 
offensive  maneuv. 

Guthrie/Pugh 

9  WP  semi-final 
draft 

5/16 

SM  306  Basic  fighter  Man. 

Princi  of  Lead  turns 
reversal  6  close-in 
maneuvers. 

Wild/Magill 

1 

12  Produce  Graphic  : 
thumbnail 

s  5/20 

FIGURE  5.2  WEEKLY  PROGRESS  REPORT 


5.3  PRODUCTION  TRACKING  CHART 


Two  types  of  Production  Tracking  Charts  were  developed  to 
indicate  the  daily  status  of  lessons.  They  were  tacked  to  the 
wall  and  designed  with  a  plastic  coversheet  that  could  be  easily 
updated  with  a  grease  pencil.  It  was  the  writer's  responsibility 
to  update  the  charts  daily. 

One  chart  listed  the  lessons  and  showed  the  current  stage  of 

production  as  of  that  day.  The  other  chart  listed  the  lessons 

and  every  step  of  the  production  process.  As  a  lesson  entered  a 

new  production  step  that  step  was  marked  with  an  X.  A  lesson 

that  had  entered  production  step  5  would  have  a  series  of  5  X's; 
once  the  lesson  had  reached  the  last  step  (USAF  acceptance)  a 
line  was  drawn  through  all  the  Xs  indicating  the  lesson  was 
finished . 
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6.0  PRE-IMPLEMENTATION  EVALUATION 

Evaluation,  both  before  and  during  implementation,  was 
critical  to  a  successful  ISD  project.  Pre-implementation 
evaluation  involved  tryout  of  all  instructional  units  by  small 
groups  of  students.  The  Student  Attitude  Questionnaire  con¬ 
tained  in  this  section  was  used  for  collecting  and  summarizing 
student's  attitudes  toward  each  lesson.  The  actual  lesson,  i.e. 
Workbook  109,  was  used  to  collect  and  summarize  student's 
comments  about  technical  accuracy  and  instructional  format. 
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6.1  STUDENT  ATTITUDE  QUESTIONNAIRE 


The  Student  Attitude  Questionnaire  was  used  to  collect  and 
summarize  student's  attitudes  towards  content,  instructional 
materials,  supplementary  materials  and  review  questions.  There 
was  also  an  opportunity  to  comment  on  the  program  as  a  whole. 
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F-16  AIRCREW  TRAINING  DEVELOPMENT  PROGRAM 
Student  Attitude  Ques t ionna i r e 


Segment  No. _ Date _ 

Segment  Title _ _ _ _ _ _ 

Place  a  check  (  )  in  the  box  which  best  describes  your  response 
to  the  following  questions: 

I.  CONTENT 

1.  This  segment: 

a  .  (  )  stands  alone 

b.  (  )  requires  interaction  with  an  instructor 

c.  (  )  requires  supplementary  written  materials 

2.  In  relation  to  learning  about  the  F-16,  I  considered  the 
instruction  : 

a.  (  )  extremely  relevant 

b.  (  )  relevant 

c.  (  )  irrelevant 

d.  (  )  extremely  irrelevant 

3.  The  instruction  in  this  segment  was: 

a.  (  )  too  demanding 

b.  (  )  challenging 

c.  (  )  not  really  challenging 

d.  (  )  too  simple 

4.  Hie  concepts  presanted  in  this  segment  were: 

a.  (  )  clear  and  very  organized 

b.  (  )  organized 

c  .  (  )  not  organi  zed 

d.  (  )  completely  disorganized/confusing 

5.  The  material  presented  was: 

a.  (  )  extremely  interesting 

b.  {  )  interesting 

c.  (  )  somewhat  interesting 

d.  (  )  boring 


FIGURE  6.1  STUDENT  ATTITUDE  QUESTIONNAIRE 


The 

mate 

rial 

presen  ted 

in 

the 

segmen  t 

conta ined : 

a  . 

( 

) 

the 

right  amo 

un  t 

o  f 

informat: 

ion  for 

the  time 

a  1 lotted  . 

b. 

( 

) 

too 

much  info 

rmation 

for  the 

time  a] 

L lotted . 

c  . 

( 

) 

too 

little  in 

formati 

on  for  the  time 

allotted 

II.  INSTRUCTIONAL  MATERIALS 

7.  The  choice  of  language  and  style  was: 

a.  (  )  very  appropriate  for  the  intended  audience 

b.  (  )  appropriate  for  the  intended  audience 

c.  (  )  inappropriate  for  the  intended  audience 

If  you  checked  c,  explain  why  _ _ 


8.  The  illustrations  were; 


a . 

(  ) 

always 

technically  correct 

b . 

(  ) 

usually 

technically  correct 

c . 

(  ) 

not  at 

all  technically  correct 

If  you  checked  b  or  c,  was  variance  from  technical  accuracy 
confusing  to  you?  Explain 


9.  The  ill  ustrations  : 

a.  (  )  greatly  enhanced  my  understanding  of  the  content 

b.  (  )  enhanced  by  understanding  of  the  content 

c.  (  )  did  not  enhance  my  understanding  of  the  content 

If  you  checked  c,  explain  why  _ _ ______ 


III  SUPPLEMENTARY  MATERIALS  AND  REVIEW  QUESTIONS 

10.  Hie  supplementary  worksheet  and  charts  were: 

a .  (  )  very  useful 

b.  (  )  useful 

c.  (  )  somewhat  useful 

d.  (  )  not  at  all  useful 


FIGURE  6.1  STUDENT  ATTITUDE  QUESTIONNAIRE 
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11.  The  review  test: 


a.  (  )  covered  only  the  instructional  materials 

b.  (  )  covered  material  not  presented  in  the  instruction 

12.  The  review  questions  emphasized: 


a  . 

(  ) 

info rma t  ion 

very  important  to  my  job 

b . 

(  ) 

in  fo  rmation 

important  to  my  job 

c  . 

(  ) 

some  useful 

information 

d. 

(  ) 

tr iv  ial/un r 

elated  information 

13.  Comment  on  the  program  as  a  whole.  Point  out  areas  which 

could  be  changed  to  improve  the  effectiveness  of  the  segment 


FIGURE  6.1  STUDENT  ATTITUDE  QUESTIONNAIRE 


7.0  DATA  BASE  UPDATE 


There  were  a  number  of  different  data  bases  that  required 
continual  update  and  revision.  They  included  the  pilot  task 
analysis,  goal  analysis,  CROs ,  objectives  hierarchies,  and  program 
reports.  Since  changes  to  one  data  base  often  times  has  implica¬ 
tions  for  other  data  bases,  a  systematic  procedure  for  making  these 
changes  is  required.  The  following  forms  were  designed  to 
facilitate  this  process. 


7.1  Task  Revision  Form 

The  task  revision  form  was  used  to  document  the  proposed 
revisions  to  the  pilot  task  analysis,  objectives  hierarchies,  goal 
analysis,  and  criterion- referenced  objectives  (CROs).  The  form 
provided  a  means  for  establishing  the  history,  rationale,  and  im¬ 
pact  of  a  suggested  change. 

The  use  the  forms,  the  author  fills  in  the  date  and  his  name 
or  initials.  He  then  determines  whether  the  change  affects  the 
task  listing,  CROs,  or  objectives  hierarchies  or  any  combination  of 
these,  and  checks  the  appropriate  column  under  "Change  to:".  Under 
the  "Action  to  Task"  heading,  he  checks  whether  the  change  is  an 
addition  (A)  of  a  new  task  or  a  deletion  (D)  or  modification  ( M)  of 
an  existing  task.  If  the  task  already  exists,  its  number  is  re¬ 
corded.  If  the  task  is  being  added,  there  may  not  be  any  task 
numbers  unless  placement  in  the  task  listing  has  been  determined. 
(Renumbering  will  be  accomplished  later  by  the  data  base  manager) . 

A  proper  entry  in  the  "Description  of  Change"  column  should  ensure 
that  the  changes  are  accurately  described.  For  a  new  task  or  a 
task  being  changed,  the  latest  form  of  the  description  is  entered. 
The  "Reason  for  Change"  column  is  completed  next.  A  preliminary 
indication  is  then  made  of  which  aircraft  systems  are  likely  to  be 
affected  by  the  change.  This  process  is  repeated  for  each  change 
input . 
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TASK  REVISION  FORM 


7.2  Report  Revision  Form 


The  F-l 6  training  project  developed  a  series  of  thirty-five 
reports.  These  reports  were  changed  on  an  as  needed  basis  to 
reflect  new  information. 

The  change  author  fills  in  the  date  and  his  name  or  initials 
on  the  Report  Revision  Form.  Next,  he  enters  the  title/number  of 
the  report  being  changed.  He  then  determines  how  the  report  is 
af f ected--whether  the  change  is  an  addition,  deletion,  or 
modification — and  checks  the  appropriate  box  under  the  "Action" 
column.  In  the  next  column  he  enters  a  description  of  the  change 
and  then  describes  the  reason  for  the  change  in  the  next  column. 
This  process  is  repeated  for  each  change  for  a  particular  stored 
project  report. 


REPORT  REVISION  FORM 


FIGURE  7.2  REPORT  REVISION  PORN 
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Figure  2 — Report  Revision  Form 


